2 research outputs found
On the Difference of BERT-style and CLIP-style Text Encoders
Masked language modeling (MLM) has been one of the most popular pretraining
recipes in natural language processing, e.g., BERT, one of the representative
models. Recently, contrastive language-image pretraining (CLIP) has also
attracted attention, especially its vision models that achieve excellent
performance on a broad range of vision tasks. However, few studies are
dedicated to studying the text encoders learned by CLIP. In this paper, we
analyze the difference between BERT-style and CLIP-style text encoders from
three experiments: (i) general text understanding, (ii) vision-centric text
understanding, and (iii) text-to-image generation. Experimental analyses show
that although CLIP-style text encoders underperform BERT-style ones for general
text understanding tasks, they are equipped with a unique ability, i.e.,
synesthesia, for the cross-modal association, which is more similar to the
senses of humans.Comment: Natural Language Processing. 10 pages, 1 figure. Findings of ACL-202
CMB: A Comprehensive Medical Benchmark in Chinese
Large Language Models (LLMs) provide a possibility to make a great
breakthrough in medicine. The establishment of a standardized medical benchmark
becomes a fundamental cornerstone to measure progression. However, medical
environments in different regions have their local characteristics, e.g., the
ubiquity and significance of traditional Chinese medicine within China.
Therefore, merely translating English-based medical evaluation may result in
\textit{contextual incongruities} to a local region. To solve the issue, we
propose a localized medical benchmark called CMB, a Comprehensive Medical
Benchmark in Chinese, designed and rooted entirely within the native Chinese
linguistic and cultural framework. While traditional Chinese medicine is
integral to this evaluation, it does not constitute its entirety. Using this
benchmark, we have evaluated several prominent large-scale LLMs, including
ChatGPT, GPT-4, dedicated Chinese LLMs, and LLMs specialized in the medical
domain. It is worth noting that our benchmark is not devised as a leaderboard
competition but as an instrument for self-assessment of model advancements. We
hope this benchmark could facilitate the widespread adoption and enhancement of
medical LLMs within China. Check details in
\url{https://cmedbenchmark.llmzoo.com/}